UNITARY REPRESENTATIONS AND HEISENBERG PARABOLIC SUBGROUP 

HONGYU HE 

C " !) , Abstract. In this paper, we study the restriction of an irreducible unitary representation tt of 

Cn ' the universal covering 5p2„(R) to a Heisenberg maximal parabolic group P. We prove that if 7r|p 

J, ' is irreducible, then tt must be a highest weight module or a lowest weight module. This is in sharp 

C^ ' constrast with the GLni^) case. In addition, we show that for a unitary highest or lowest weight 

module, Trjp decomposes discretely. We also treat the groups U{p,q) and 0*(2n). 

(N 

1. Introduction 

rj , Let F = C,R. Let G'L„(F) be the general linear group on F". Let Pi be the maxmial parabolic 

Pm ' subgroup preserving a one dimensional subspace in F". Let n be an irreducible unitary representa- 

r~| , tion of GL„(F). Consider the restriction of n onto Pi. Kirillov conjectured that ttJpj is irreducible. 

Kirillov's conjecture was proved by Sahi using Vogan's classification ( |Sah) |Vogan| ) . Recently, 

Baruch established Kirillov's conjecture without Vogan's classification ( |Baru] V 
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Generally speaking, for other semisimple Lie groups G, the restriction of an irreducible unitary 
04 ' representation of G to a maximal parabolic subgroup is hardly irreducible. Nevertheless, as proved 

by Howe and Li, for irreducible low rank representations, their restrictions to a certain maximal 
parabolic subgroup remain irreducible ( [Howes] (Li)). In this situation, the restriction uniquely de- 
fvq , termines the original representation. However, it is not clear whether there are other representations 

ly^ ■ whose restriction to a fixed maximal parabolic group is irreducible. 

(•^ ■ Now Let G = Sp2n{^) and n > 2. Let P be the maximal parabolic subgroup that preserves a 

^^ i one- dimensional isotropic subspace of the symplectic space K^". Decompose the identity component 

of P as Sp2n-2{^)AHn-i whcrc iJ„_i is the Heisenberg group and A = M+. We call P a Heisen- 
berg parabolic subgroup of G. Let G be the universal covering of G. For any subgroup H of G, 

L^ ■ let H be the preimage of H under the universal covering. The classification of irreducible unitary 

i*^ . representation of P can be obtained directly by Mackey analysis. 

As an example, take the linear group P. Let G(i/„_i) be the center of Hn-i- Let tt be an ir- 
reducible unitary representation of P. If T^\c(Hn-i) is trivial, then tt is in one-to-one correspondence 
with irreducible unitary representations of maximal parabolic subgroup of S'p2n-2(K) with levi fac- 
tor GLi{R)Sp2n-A{^)- Suppose 'r^\c{H,^-i) is not trivial. Let p± be the two irreducible unitary 
representation of dilated Heisenberg group AHn-i- Then P is parametrized by a triple (p±,r, ±) 
where r is a genuine irreducible unitary representation of Mp2n-2(K) and ± corresponds to the 
two representations of the component group of P . Notice that p± can be extended to a unitary 
representation of Mp2n-2(R), and r can also be extended trivially to AHn-i- Every irreducible 
unitary representation of P can be written as, p± ® t ® C± . All tensor product of Hilbert spaces in 
this paper will mean the completion of the algebraic tensor product. 
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For simplicity, let us absorb the parameter ± into t. Any unitary representation tt of P can be 
written as 

[p+(g)r+]® [p_«)T_]©ro, 

here To|c(ff„_i) i^ trivial. Hence every irreducible unitary representation of G can also be written 
in this form. Now p+|A/p2n-2(R) ^^ equivalent to Lo{n — 1) ® C°° where uj{n ~ 1) is the oscillator 
representation and C°° is an infinite dimensional trivial representation of Mp2n-2(M)- P-|Mp2re-2(R) 
is equivalent to w(n — 1)* (S) C°°. 

Theorem 1.1 (See [Howej ) . Let n be a nontrivial irreducible unitary representation o/ S'pjn (I^) • 
Then there are two unitary representations T_|-(7r) and T„(7r) of Sp2n-20^) such that 

^Po = [P+ ® T+i^)] ® [P- ® T-i^)], 
^lsp,„_,(R) - [^ ® ^+ W ® ^* ® ^-(^)] ® '^°°- 

This theorem is established by Howe for the double covering of Sp2n(^) ( [Howej ) . Howe's argu- 
ment essentially extends to the universal covering of Sp2n (K) • 

If TT is a unitarily induced representation from a unitary representation of P, then t+ and r_ are 
quite easy to compute. The issue of computing the map tt ^- r± (tt) for smaller representations is 
rather complex. For the constituent of the oscillator representation oj+, r+ is trivial and r_ is zero. 
For the constituent of the oscillator representation w_, t+ is the sign character of GLi{M.) and r_ is 
zero. In this paper, we give some results concerning t±{tt). By Mackey analysis, 7r|p is irreducible, 
if and only if one of t± (tt) vanishes and the other is irreducible. 

Theorem 1.2. Let n be an irreducible unitary representation of G. If Tr\p is irreducible, then tt 
must be either a highest weight module or lowest weight module. In addition, for tt a unitary lowest 
weight module, 7r|p ^ /?+ ''"-(-(tt) where T_|-(7r) decompose discretely into a direct sum of lowest 
weight modules of Sp2n-2i^); for tt a unitary highest weight module, 7r|p = p_ (X)r_(7r) where t-{tt) 
decompose discretely into a direct sum of highest weight modules of Sp2n-2(M-)- 

It is not clear whether t±{'k) is irreducible for n a highest or lowest weight module. For some 
highest (lowest) weight modules, t±{tt) is irreducible. In fact, for Sp2{R), t±{tt) will always be irre- 
ducible. For n > 2, decomposing 7r|p is quite difficult, because 7r|p does not decompose according 
to the ii'-types. 

In this paper, we derive some equivalent conditions for tt being a nontrivial highest weight module. 
One of the condition can be stated as follows. 

Theorem 1.3. Let 5^2(1^) be a subgroup of G that fixes a nondegenerate 2n — 2 dimensional sym- 
plectic subspace. Let N be a unipotent subgroup in Sp2i^)- Identify N with the real line. Then tt is 
a nontrivial irreducible unitary highest weight module if and only if tt\n is supported on half of the 
real line. 

In this paper, we also treat the groups U{p,q) and 0*{2n). The group P will be a maximal 
subgroup whose nilradical is a Heisenberg group. We call such P a Heisenberg parabolic subgroup. 
The detailed results are stated in Theorems 16.11 16. 2[ 17. 1[ 17.21 
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2. Irreducible Unitary Representations of P 

Let G be the symplectic group Sp2n{^) with n > 2 and P be the maximal parabohc subgroup 
preserving a one dimensional isotropic subspacc Mei. Let G be the universal covering of G. For 
simplicity, let Z be the preimage of the identity. The group P has a Langlands decomposition 
GLi{E.)Sp2n-2{^)Hn-i whcrc Hn~i is the Heisenberg group. P is a semidirect product of GLi(M) x 
Sp2n-2{^) and Hn-i- GLiiK) can be further decomposed as Z2A with A = M"*". 

Lemma 2.1. A = ZAq where Aq is the identity component of A which can be identified with A. So 
we will write A = ZA. In addition, GLi(M.) ^ ^IjA. Lastly 

P9^izA§p2„_2(R)i7„_i/Z. 

So 7ro(P) = Z2. Here GLi{R) n 5p2«-2(K) = Z. 

Notice here that the adjoint action of GLi (K) on -ff„_i descends into the adjoint action of GLi (M) 
on Hn-i and the adjoint action of S'p2n-2(^) descends into the adjoint action of 5p2n-2(K) on ifn-i- 

Suppose that A is real and A 7^ 0. Let px be the unique irreducible unitary representation of 
Hn-i with central character expiAi. The adjoint action of G'Li(R) on Hn-i induces an action of 
GLi(R) on Hn-i- In particular, ±1 e G'ii(R) perserve p\ and 

a e GLi(M) : px -^ Pa^x- 

By Mackey analysis, there are two irreducible unitary representations of AHn-i: 

p+ ^ pxd\, P- ^ PAC^A. 

J \eTSL+ J xeTSL- 

These are the only irreducible unitary representations with /9|c(ff„_i) 7^ I, the identity. Now ±1 € 
GLi(]R) preserves each px- By Stone- Von Neumannn Theorem, ±1 acts on each px projectively. In 
this situation, it is easy to make ±1 act on px directly. There is no obstruction to lift the projective 
action of ±1 on p^- Using the Schrodinger model, —1 acts on the odd functions by —1 and on the even 
functions by +1. Let us include the actions of Z2 C GLi(R) in the model px, consequently in p±. 
Now again, by Mackey analysis, there are four irreducible unitary representations of GLi(R)if„_i 
on which G(-ff„_i) acts nontrivially, namely 

p±(E)Sgn, p±. 

The difference between the former and the latter is a little subtle. One way to tell the difference 
is that p± ® sgn(— 1) acts on the even functions by —1 while /3±(— 1) acts on the even functions by 
identity. 

Now consider GLi(M)iJ„_i. The representation p± can be regarded as a representation of Gii(R)-ff„_i. 

Lemma 2.2. Identify Gii(R)i7„_i with ^Z AHn-i- Then the irreducible unitary representations 
on which G(iJ„_i) act nontrivially are all of the form 

P±®Xt\t^ [0, 1) 

with Xti^n) — expAnimt for m G ^Z. 

For GLi(R)i7„_i, t = Q,\ because xt(Z) = 1. 

Now let us consider Sp2n-2(^)- This group preserves px- Again, by the Stone-Von Neumann 
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Theorem, Sp2n-2{^) f^cts on px projcctively. Since Sp2n^2(M) is already simply connected, one 
obtains a group action of S'P2n-2(^) '^^ Pa- By a theorem of Segal-Shale- Weil, Sp2n~2i^) action on 
p\ descends into an action of Mp2n-2{^)- Simply put, m ^Ti acts by (—1)™ — expirmr on px. We 
can now extend p± to include the action of 5p2n-2(-'^)- 

From now on p± will be representations of P. 

Theorem 2.1 (Howe). Irreducible unitary representations of P on which C(i?„_i) acts nontrivially 
are of the form 

[P± <^Xt]<»T 

with T an irreducible unitary representation of Sp2n-2^) such that T{m) — Xti'in) for any to G Z. 
In addition, two such representations are equivalent if and only if all the parameters (±,t, r) are the 
same. 

I shall make some remarks here. First, r is extended as a representation of P, trivially on AHn~i, 
amd trivially on the component group. Second, Xt 'S!) t is a twisted tensor product in the sense that 
the action of Z commutes with the tensor. For group P, p±{m) ® Xt("^) ® T{m) must be the identity 
for every to € Z C G. So (— 1)"' exp47r?7it = (— 1)'"t(to.) — 1. For an irreducible unitary represen- 
tation of P on which C{Hn-i) acts nontrivially, i = ^, | and r is a genuine unitary representation 

0fAfp2n-2(M). 

The proof is straight forward by applying the Mackey analysis. Observe that the subgroup of 
P that preserves p± is ^Z x^ Sp2n-2(^) ■ Xt® t parametrizes the equivalence classes of irreducible 
unitary representations of this subgroup. 

3. Irreducible Unitary Representations of 5^2(1^) 

Throughout this section G = S'p2(R) and P = MAN where M = Z2 and A = R+ and A^ ^ M. 
What we have proved in the last section needs to be modified. Since the unitary dual of G is known 
( |Puk) . [HT] ) . we will analyze 7r|p in detail. The results in this section must have been known to 
the experts. Some of the result will be used in the next section to analyze higher rank case. 

There are essentally four classes of irreducible unitary representations of G (see |Puk) ) : 

(1) the trivial representation 1; 

(2) unitary principal series J(e, s) where e G [0, 1) and s G iM ( we exclude e = i, s = 0); 

(3) complementary series C(e, s) (e S [0, 1), s G (0, |1 — 2e|)); 

(4) Highest weight modules D'j^ {I > 0) and lowest weight modules D^{1 > 0). 

Let P be the standard parabolic subgroup of G. Let N be the nilradical of P. Then the identity 
component Pq has two irreducible unitary representations on which N acts nontrivially, namely p+ 
and /9_ . p+ Itv is supported on K+ C iV, and P-\n is supported on K^ C N. Now the center of G can 
be identified with ^Z. Identifying P with ^ZAN, p± can be extended to a representation of P by 

identify Pq with P/^Z. Then elements of P are parametrized by (±, t) with t £ [0, 1). More precisely, 
every irreducible unitary representation of P is equivalent to p± iSi Xt- Here xt (man) — exp Animt 
with TO € iZ, a G A and n G N. Notice that Xt also defines a central character of G. In our 
setting, the representations with even weights have central character xo; the representations with 
odd weights have central character Xi- 
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The following theorem gives the structure of the restriction of irreducible unitary representations 
of G to P. It is known to the experts. However, I could not find an elementary proof in the literature. 

Theorem 3.1. 

(1) I{e,s)\p'^(p+®P-)®Xe; 

(2) C{e,s)\p = {p+(S)p-)®Xe\ 

(3) D+\p'^p+®xi-y^\] 

(4) -D; Ip = P- ® Xi_i+|i|- Here [^J is the largest integer less or equal to ^. 

Proof: The central character of each tt e G can be computed easily. Using the noncompact 
model, /(e, s) can be modeled on Lp'{N) with N act as translations. Hence (1) is proved. To prove 
(2), (3), (4), it suffices to show that the Fourier transform of the matrix coefficients of tt restricted 
to N has the desired support. 

To show (2), let (, ) be the inner product of C{e,s) and (, )/„d be the natural pairing between 
the induced representations /°°(e, — s) and /°°(e, s). For smooth vectors 0, V' G I°°{^is), we have 

(</«,-0) = (A(e,s)0,V')/«d 

where A(e, s) is the intertwining operator defined over smooth vectors. In addition, A{e, s) defines 
a bijection between /°°(e, s) and /°°(e, — s). Using the noncompact model, for every n e iV as an 
additive group, we have 

(C(e,s)(n)(/), ■(/;) = (A(e,s)0,/(e,s) (-n)?A)/„d == / A{e,s)(t)(x)ip{x - n)dx. 

JN 

Now C^iN) C /°°(e,±s). We choose and V^ so that 

A(e, s)(t) e C^{N), ^ e C^{N). 

So (C(e, s){n)(j)^ -0) becomes the convolution of two smooth and compactly supported functions. Its 
Fourier transform can be made to be supported on N , upon proper choices of (j) and ^. So again we 
have C(e, s)\p ^ {p+ ® p_) (g) Xe- 

We will now prove (3). (4) follows immediately from (3). Notice that D'^{1 > 0) is a subquo- 
tient of /(e, I — 1) and of /(e, 1 — with e = | — L^J • Let vi+p be of weight ^ +p in D^^ where p is a 
nonnegative even integer. We stick with the noncompact picture. Let 

, 1 .iA+xii±p {1 + xi)2 

'Pi+p - ( 7 9,-, ) ll ■) ^ - T, ^j+e" 

Va;2 + 1 1 - a;* (1 - xiy+^ 

be a function in the noncompact model of I{e,2l — I). Here we choose the standard arg funtion 
between — ^ and ^ to define (1 ± x«)' ii I is not an integer. Let 

be a function in the noncompact model of /(e, I — 21). Then 

{D+{g)vi+p,vi+,) = C{I{e,2l-l){g)^i+p,iji+g) {geG,p,q> 0). 
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In particular, for n G iV, 

(1) 
Notice that 



{DJ{n)vi+p, vi+q) = I (j)i+p{x - n)^i+q{x)dx 



N 



_ (expiO0i+p(^)V'i+g(C)c^? 



N 



{1-xi) 



1+ 



— — Ci 






So the Fourier transform of ^ j, is supported on R+ (See Ch. 8.3 jFo] ) . Hence the Fourier 

{l—xi) 2 
p 

transform of (/>/+„ (a;) = -^^ — ^', fi is also supported on M+. Even though ipij^qlx) are not in L"'^, 

{l~xi) ^ 

-l^-[T[-^:3i'l+q{x) will be in L^ . So Fourier transform of ^;+g(x) will be a Co(K) function multi- 
plied by a monomial of ^. Hence, Eq. [T] holds. We have thus seen that the Fourier transform of 
{D'^ {n)vi+p,vi+q) is supported on M+. Since {w;+p | p > 0} is an orthogonal basis for D'^ , D'^\j^ is 
supported on R+. 

I shall remark that there may be a more "topological " proof of Theorem 13.11 The proof of (3) 
and (4) I give here is more self-contained. 

4. Restriction of Unitary Representations and Irreducibility 

Now let TT be a nontrivial irreducible unitary representation of G. Then 7r|p can be decomposed 
into a direct integral of p± 'Si Xt ® "T- In particular, one can write 

TT = /9+ (g) T+(7r) ® p_ (X) T_(7r) ® Tq. 

Here T+(7r) and T_(7r) are unitary representations of GLi{M)Sp2n-2(^)- 

Theorem 4.1 (See |Howe| ) ■ Let t: be a nontrivial irreducible unitary representation of G. Then 
there exist two unitary representations T+(7r) and T_(7r) of GLi{]&)Sp2n-2i^) such that 

7r|p = p+ (8)T+(7r) ® p_ (8)T_(7r). 

Notice that one of t±{tt) could be zero. This theorem was proved by Howe in [Howe] Pg. 249 for 
the metaplectic group. 

Proof: We will have to prove that tt\p does not have any subrepresentation on which C(i?„_i) 
acts trivially. Let u be a nonzero vector fixed by C(7f„_i). Let Gq be the subgroup of G that com- 
mutes with Sp2„-2W- So Go ^ 5^2(K). Notice that C(i?„_i) C Gq and A C Gq. Let H be the 
Hilbert space spanned by gov iovgo € Go ■ Clearly, "H decomposes into a direct integral of irreducible 
unitary representations of Gq on which Z acts as a character. Indeed, all factorial representations 
of Go are direct sum of irreducible representations. Now let 

Vsdfi{s) 

Go 

where Vg € Hs <E) Vs, Hs G Gq and dimT4 — m{'Hs^'H). Then G(if„_i) must fix Vs for almost all s 
with respect to ji. If T-Ls is not trivial, T-Ls ® Vg has no vector fixed by G(-ff„_i). Hence, % must be a 
direct sum of the trivial representation of Go. In particular, tt must descend to a representation of 
G. The matrix coefficent g — > {n{g)v,v) violates the Howe-Moore vanishing Theorem ( [HMj ) . We 
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reach a contradiction. D 



Theorem 4.2. Let tt be a nontrivial irreducible unitary representation of G such that Z acts by 
exp2'Kmt (Vto G Zj for a fixed t £ [0, 1). Suppose that 7r|p is irreducible. Then n must be a highest 
weight module or a lowest weight module. 

Proof: Let us fix the standard maximal compact group U{n) C G. Then U{n) D S'p2n-2(K) = 
f/(n — 1). As usual, the complexified Lie algebra gc decomposes into a direct sum 

Suppose that 7r|p is irreducible. By the last Corollary, either t^{tt) or T„(7r) must be zero. Without 
loss of generality, suppose that 7r|p = p+ (g) T^{n). Notice that p+\c{H„-i) is supported on R+. 
So 7t\c(h„^i) rnust also be supported on R+. Let Gq be the subgroup of G that commutes with 
Sp2n-2{^)- Consider the restriction 7r|g, . 7r|g, can be decomposed into a direct integral of irre- 
ducible unitary representations with multiplicities. By Theorem 13. 1[ among the irreducible unitary 
representations of Gq, only the lowest weight modules are supported on M+ C C{H„-i)- Hence only 
the lowest weight modules occur in the direct integral decomposition of -Kq . 

Now 7r|(5 is a direct integral of lowest weight modules. Let [/(I) = Gq n U{n). Then Trl^,.^^ 
can only have positive weights. Fix a maximal torus T in U{n) C G. Then 7r|j, can only have 
positive weights. Let v\ be a vector with weight A such that ^ Aj is minimal among all possible 
weights occuring in ■n\f. Notice that ^ A^ must be a nonnegative integer. So a minimal ^ A^ must 
exist. Now 7r|^. s must contain an irreducible representation V^ with "^ iJn —^\i. Clearly, p~ act 
on Vfj, by zero. So the module generated by V^ must have a lowest weight module as its quotient. 
Since tt is already unitary and irreducible, tt must be a lowest weight module. Now we have shown 
that TT is a unitary lowest weight module. D 

I shall remark that the last paragraph is true even one assumes that the weights for 7r|^^% is 
bounded from below. 

5. Some Criterions for Lowest Weight Modules 

In this section, we give some characterization of lowest weight modules in terms of their restric- 
tions on certain subgroups. Some of them are well-known to the experts. Let us fix a complex 
structure and an inner product ( , ) on the symplectic space M'^" such that the symplectic form 
coincides with the imaginary part of ( , ). Let ei, e2, . . . , e„ be the standard basis over C Let P be 
the subgroup preserving Rei. Let U{n) be the subgroup preserving ( , ). 

Theorem 5.1. Let T be a maximal torus in the maximal compact group U{n) of G. Let 5*^2(11^) 
be the subgroup of G acting on a n — 1-dimensional complex subspace by identity. Let U{1) = 
5'p2(K) n U{n). Let tt be a nontrivial irreducible unitary representation of G. Let Q be a maximal 
parabolic subgroup of G. Let N be its nilradical and ZN be the center of N . The following are 
equivalent: 

(1) ■"'Izjv "is supported on the positive semidefinite cone of ZN, regarded as the space of symmetric 
matrices; 

(2) 7''lc(i?„_i) ^■^ supported on R+ C G(iJ„_i); 

(3) 7r|g- c^N decomposes into a direct integral of lowest weight modules; 
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(4) 7r|^^% only has positive weights; 

(5) 7r|j. only has positive weights; 

(6) the weights o/7r|^/-^-, is bounded from below; 

(7) there is an integer k such that every weight A of TT\f satisfies Xi > k for every i; 

(8) TT is a unitary lowest weight module of G; 

(9) If n — I, TT is a unitary lowest weight module; if n > 2, 7r|p decomposes into p+ T-|-(7r) 
and that the weights o/ r-|_(7r)|jy^s are bounded from below. Here U{1) is a subgroup of the 
Sp2n~20^) factor of P. 

Proof: When n = 1, our theorem foUows from Theorem 13. II Suppose now n >2. 

By [Howej . Tr\zN is supported on GL-orbits on ZN. (1) o (2) is a matter of matrix analysis. 
(2) — > (3) — > (4) is proved in Theorem 14.21 (4) — > (3) — > (2) is easier than the other direction. So 
(1), (2), (3) and (4) are equivalent. 

(4) -^ (5) -^ (6) -^ (7) is trivial. (7) -^ (8) follows as in Theorem O (8) -^ (6) is also obvi- 
ous. So (8), (7), (6) are equivalent. 

To prove (8) — ?► (9), suppose tt is a nontrivial unitary lowest weight module. By Cor. 14.11 ^\p — 
p+ (g) T+(7r) © p_ (g) T_(7r). If r_(7r) ^ 0, fix a C/(l)-eigenvector v with eigenvalue Ai. By tensoring 
with vectors in p„, we obtain [/(l)-eigenvector with arbitarily low eigenvalue. This contradicts (6). 
So TT = p+ (8)r+(7r). Similarly, weights for T+{T^)\ijti\ must also be bounded below. Hence (8) — > (9). 

Suppose that (9) holds. So the weights of t+{t^)\ijii\ are bounded from below. This implies that 
T-f (7r)|s- ,„, must be a direct integral of lowest weight modules with multiplicites. So the weights 
of T+(7r)|jy^-, must all be positive. Hence the weights oiir]^,^-. must all be positive. Thus (9) -^ (4). 
We have proved (9) -^ (4) -^ (5) -^ (6) -^ (7) -^ (8) ^ (9). D 

I shall remark here that the parametrization of unitary highest or lowest weight modules is al- 
ready known, due to the work of Enright-Howe-Wallach ( |EHWj l. It is perhaps easy to go further 
to derive more properties of unitary lowest or highest weight modules from Theorem 15. II 

Corollary 5.1. Let tt be an irreducible unitary lowest weight module of G. Then 7r|p = p+ 0T+(7r) 
and T-f-{TT) decomposes into a direct sum of lowest weight modules of Sp2n-2^)- 

Proof: By Theorem 15.11 (9), the weights of t+{t^)\ijii) are bounded from below. By Theorem 
15.11 (4), T+(7r)|^^x only has positive weights. By Theorem 15.11 (7) (8), there is a lowest weight 

subrepresentation of 5'p2n-2('^) i^i t^{tt). Consider the orthogonal complement. If it is nonzero, 
then there is another lowest weight subrepresentation. This process can continue and it will end in 
countable time due to the fact that tt has a countable basis. We now obtain a discrete decomposition. 

D 

6. The group U{p,q) 

Suppose p > q > 1 and p + q > S. Let U{p, q) be the group that preserve a Hermitian form ( , ) on 
CP+9 with signature (p, q) . Let P be a maximal parabolic subgroup that preserves a one dimensional 
isotropic subspace. Then P can be identified with GLi{C)U{p — l,q— l)iJp+q_2- Here i/p+g_2 are 
parametrized by (i G R, m S C^"'"''"^) . The adjoint action of g G U{p — 1, g — 1) on Hp^q-2 leaves 
t fixed and operates on u as the left multiplication. The adjoint action of a £ GLi(C) on Hp+q-2 
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dilates t to ||a|pi and operates on u as scalar multiplication. Write GLi{€.) = AU{1) where A ~ M+. 

Let G — {{g,t) \ g (z U{p,q),exp2T:it — detg} be an infinite covering of G. For any subgroup 
H of G, let H be the preimage of H. Let p± be the irreducible unitary representations of AHp+q-2 
we defined earlier. These are the only irreducible unitary representations on which G{Hp+q^2) acts 
nontrivially. Extend p± to a unitary representation of P. Notice that P+\i}(i)(j(p^i „_i) decomposes 
as follows 

[®„gz;det ^ (g) 6'(det ^)] (g) C°°. 

Here det 2" is a character of f/(l), 9{dei 2") is the theta lift of det 2" with respect to (C/(l), U{p — 
l,g — 1)) (see [H]) and C°° records the multiplicity. 

Theorem 6.1. Let tt he an infinite dimensional irreducible unitary representation ofG. Then there 
are two unitary representation r±(7r) ofU{p— l,g— 1) such that 

Ap — P+ ®t+{-k) ® P- (X)T_(7r). 

Let U{1) be diagonally embedded in C/(l, 1) C U{p,q). Then 

Mu{i)u(p-i,q-i) - [©nezdet* ®6'(det^)]®T+(7r)®C°°® [®„gz;det^ ® 6l'(det ^)] «) T_(7r) «) C°°. 

Here 9' refers to the theta lifts with respect to the congredient oscillator representation. 

Notice that one of the t±['k) could be zero. Similarly, we can prove 

Theorem 6.2. Let vr he an infinite dimensional irreducible unitary representation of G. Suppose 
that 7r|p is irreducible. Then n must be a highest weight module or a lowest weight module. 

Essentially, SU{1, 1) in SU{p,q) will play the role of 5p2(R) in Sp2n(^)- The proof is omitted 
here. 

7. The group 0*(2n) 

Let n > 3. Let 0*{2n) be the group of isometry preserving a nondegenerate skew-Hermitian 
form on H". Let P be the maximal parabolic subgroup preserving a 1 dimensional isotropic sub- 
space. Then P can be identified with GLi{M)0*{2n — 4)H2n-4 where H271-4 is a Heisenberg group 
parametrized by 

(teK,ueH"-2)_ 

GLi(H) can be further decomposed as 5*^(1)^ where A is the center of GLi(M). The adjoint action 
of a e A on H2n-i is given by 

{t, u) — ?► (a^t, au). 

The adjoint action of 0*{2n — 4) on H2n-A is the left multiplication on u. The adjoint action of 
k E Sp{l) on i?2n-4 is the right multiplication. Clearly, Sp{l) x 0*{2n — 4) action preserves the 
real part of canonical skew-Hermitian form on H"^^. {Sp{l), 0*{2n — 4)) becomes a dual reductive 
pair (See 0]). 

Now let p± be the two irreducible unitary representations of AH2n-i on which C{H2n-4) acts 
nontrivially. p± extends to irreducible unitary representations of the linear group P. In particular, 
p+\sp(i)o* (2n-A) decomposes according to the theta correspondence with infinite multiplicity: 



®.es;(I)'^^^(^)^C° 
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Similarly, 

P-|5p(l)0*(2n-4) = ®,6s;(I)^ ® ^' i'^) ® C°° . 

Here 9' is the theta correspondence with respect to the contragredient oscillator representation. 
Sp{l) is parametrized by N. 

Theorem 7.1. Let tt be a nontrivial irreducible unitary representation of 0*{2n). Then there exists 
two unitary representations t±(t:) of 0*(2n — 4) such that 

tt\p = p+ 0T+(7r) ©p_ 0T_(7r). 

In additon, 

Asp(i)0' (2„-4) = {®,eSMi) ["^ ® ^("^^1 ® ^+ ^^) ® C°°} ® {©,^5^ k ® 0'(a)] <» t_ (tt) ® C°°} 

One of t±{tt) could be zero. The theorem for the universal covering of 0*{2n — 4) is left to the 
reader. Similarly, we have 

Theorem 7.2. Let tt be a nontrivial irreducible unitary representation ofO*{2n). If tt\p is irre- 
ducible, then n must be a highest weight module or lowest weight module. 

Notice that the group 0*(4) contains a noncompact factor S'L2(M). The proof is essentially the 
same as in Theorem 14.21 
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UNITARY REPRESENTATIONS AND HEISENBERG PARABOLIC SUBGROUP 

HONGYU HE 

C " !) , Abstract. In this paper, we study the restriction of an irreducible unitary representation tt of 

Cn ' the universal covering 5p2„(R) to a Heisenberg maximal parabolic group P. We prove that if 7r|p 

J, ' is irreducible, then tt must be a highest weight module or a lowest weight module. This is in sharp 

C^ ' constrast with the GLni^) case. In addition, we show that for a unitary highest or lowest weight 

module, Trjp decomposes discretely. We also treat the groups U{p,q) and 0*(2n). 

(N 

1. Introduction 

rj , Let F = C,R. Let G'L„(F) be the general linear group on F". Let Pi be the maxmial parabolic 

Pm ' subgroup preserving a one dimensional subspace in F". Let n be an irreducible unitary representa- 

r~| , tion of GL„(F). Consider the restriction of n onto Pi. Kirillov conjectured that ttJpj is irreducible. 

Kirillov's conjecture was proved by Sahi using Vogan's classification ( |Sah) |Vogan| ) . Recently, 

Baruch established Kirillov's conjecture without Vogan's classification ( |Baru] V 



C^ 



> 



Generally speaking, for other semisimple Lie groups G, the restriction of an irreducible unitary 
04 ' representation of G to a maximal parabolic subgroup is hardly irreducible. Nevertheless, as proved 

by Howe and Li, for irreducible low rank representations, their restrictions to a certain maximal 
parabolic subgroup remain irreducible ( [Howes] (Li)). In this situation, the restriction uniquely de- 
fvq , termines the original representation. However, it is not clear whether there are other representations 

ly^ ■ whose restriction to a fixed maximal parabolic group is irreducible. 

(•^ ■ Now Let G = Sp2n{^) and n > 2. Let P be the maximal parabolic subgroup that preserves a 

^^ i one- dimensional isotropic subspace of the symplectic space K^". Decompose the identity component 

of P as Sp2n-20^)AHn-i whcrc -ff„-i is the Heisenberg group and A = K.+ . We call P a Heisenberg 
parabolic subgroup of G. Let G be the universal covering of G. Let Mp2n{^) be the unique double 
covering of G. For any subgroup H of G, let H be the preimage of H under the universal covering. 
The classification of irreducible unitary representation of P can be obtained directly by Mackey 



C^ ' analysis. 



As an example, take the linear group P. Let G(iJ„_i) be the center of Hn-i- Let tt be an ir- 
reducible unitary representation of P. If T^\c(Hn^i) is trivial, then tt is in one-to-one correspondence 
with irreducible unitary representations of maximal parabolic subgroup of S'p2n-2(K) with levi factor 
GLi(]R)S'p2n-4(K). Suppose T^\c{Hn-i) is not trivial. Let p± be the two irreducible unitary represen- 
tation of dilated Heisenberg group AHn-i- Then P is parametrized by a triple (pj-, r, ±) where r is 
a genuine irreducible unitary representation of Mp2n-2{!^) and ± corresponds to the two represen- 
tations of the component group of P. Extend p± to a unitary representation of Mp2n-2{J^)AHn^i, 
and extend t trivially to AHn-i- Every irreducible unitary representation of Pq can thus be written 
as p± ® T. Moreover, p±®T can be extended to an irreducible unitary representation of P. So Every 



^This research is partially supported by the NSF grant DMS 0700809 and by LSU. 

keywords; Parabolic Subgroups, Heisenberg group. Stone- Von Neumann theorem, Mackey Analysis, branching 
Formula, unitary representations, Kirillov Conjecture, Symplectic Group 



2 HONGYU HE 

irreducible unitary representation of Pq can be written as p± <Si t <^ <C± . 

In this paper, all tensor product of Hilbert spaces will mean the completion of the algebraic tensor 
product. All Hilbert spaces are assumed to be separable. We use tt to denote both the representation 
and the underlying Hilbert space. 

For simplicity, let us absorb the parameter ± into r. Any unitary representation tt of P can then 
be written as 

[p+(8)T+] ® [p_ (KiT_]©ro, 

here tq\qij^^_-^\ is trivial. Hence every irreducible unitary representation of G can also be written in 
this form. 

Notice that /5+|mp2,i-2(k) i^ equivalent to uj{n — 1) (g) C°° where uj{n — 1) is the oscillator repre- 
sentation of Mp2n-2(K) and C°° is an infinite dimensional trivial representation of Afp2n-2(K)- 
P_|mp2,i-2(k) is equivalent to uj{n — 1)* (g) C°°. 

Theorem 1.1 (See |Howe) ) ■ Let n be a nontrivial irreducible unitary representation o/ S'pjn (U^) • 
Let Pq be the identity component of P. Then there are two unitary representations T+(7r) and T-(tt) 
of Sp2n-2i^) such that 

^IS^2„-2(K) - [^('^ " ^) ® ^+('') ® ^*(" - 1) ® ^- W] ® C°°. 

Ln the first identity, t± (tt) extends trivially to representations of Pq . 

This theorem is established by Howe for the double covering Mp2n{^) ( |Howej ) . Howe's argument 
essentially extends to the universal covering of Sp2n(^)- 

If TT is a unitarily induced representation from a unitary representation of P, then t_|- and r_ are 
quite easy to compute. The issue of computing the map n — >■ t±{'k) for smaller representations is 
rather complex. For the two constituents of the oscillator representation, uj{n)±^ t+ is trivial and 
r_ is zero. Moreover, if we take into consideration of the two connected component of P, r+ for 
w(n)+ will be the trivial representation and t+ for Lo{n)- will be the sign character. 

By Mackey analysis, 7r|p is irreducible, if and only if one of t±{tt) vanishes and the other is ir- 
reducible. Our main result is the following. 

Theorem 1.2. Let n be an irreducible unitary representation of G. If Tr\p is irreducible, then tt 
must be either a highest weight module or lowest weight module. In addition, for tt a unitary lowest 
weight module, vrjp = p+ (g T+(7r) where T_^(7r) decompose discretely into a direct sum of lowest 
weight modules of Sp2n-20^)j f'^''" '^ '^ unitary highest weight module, 7r|p !^ p_ (g)T_(7r) where T_(7r) 
decompose discretely into a direct sum of highest weight modules of Sp2n-2(^)- 

It is not clear whether t±{'k) is irreducible for n a highest or lowest weight module. For some 
highest (lowest) weight modules, t±{tt) is irreducible. In fact, for 5p2(K), t±{tt) will always be irre- 
ducible. For n > 2, decomposing 7r|p is quite difficult. Generally speaking 7r|p does not decompose 
according to the K- types. 

In this paper, we derive some equivalent conditions for tt being a nontrivial highest weight module. 
One of the condition can be stated as follows. 
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Theorem 1.3. Let S'p2(]R) he a subgroup of G that fixes a nondegenerate 2n — 2 dimensional sym- 
plectic subspace. Let N be a unipotent subgroup in Sp2(^)- Identify N with the real line. Then tt 
is a nontrivial irreducible unitary highest or lowest weight module if and only i/7r|jv is supported on 
half of the real line. 

In this paper, we also treat the groups U{p,q) and 0*{2n). The group P will be a maximal 
subgroup whose nilradical is a Heisenberg group. We call such P a Heisenberg parabolic subgroup. 
The detailed results are stated in Theorems 16. 1[ 16.21 17.11 17.21 

2. Irreducible Unitary Representations of P 

Let G be the symplectic group Sp2ni^) with n > 2 and P be the maximal parabolic subgroup 
preserving a one dimensional isotropic subspace Mei. Let G be the universal covering of G. For 
simplicity, let Z be the preimage of the identity. The group P has a Langlands decomposition 
GLi(M.)Sp2n-2(^)Hn-i whcrc Hn-1 is the Heisenberg group. P is a semidircct product of GLi(M) x 
Sp2n-2{^) and Hn-i. GLi{W) can be further decomposed as 7^2 A with A = M+. 

Lemma 2.1. A = WjAq where Aq is the identity component of A which can be identified with A. So 
we will write A = lA. In addition, GI/i(M) = {^'Z)A. Lastly 

So TTo{P) = Za. Here GLi{R) D Sp2n-2{^) = ^■ 

Notice that the adjoint action of Gii(R) on iJ„_i descends into the adjoint action of GLi{M.) 
on Hn-i and the adjoint action of >S'p2n-2(^) descends into the adjoint action of S'p2n-2(R) on -ffn-i- 

Suppose that A is real and A 7^ 0. Let px be the unique irreducible unitary representation of 
Hn-i with central character expiAt. The adjoint action of GLi(R) on Hn-i induces an action of 
GLi(R) on Hn-i- In particular, ±1 e Gii(R) perserve p\ and 

a e GLi(R) : px -> Pa^x- 
By Mackey analysis, there arc two irreducible unitary representations of AHn-i. 

P+= I P\d\, P- ^ I P>^^^- 

These are the only irreducible unitary representations with p|c(ff„_i) 7^ I, the identity. Now ±1 G 
GLi(IR) preserves each px. By Stone- Von Neumannn Theorem, ±1 acts on each px projectively. In 
this situation, it is easy to make ±1 act on px directly. There is no obstruction to lift the projective 
action of ±1 on p^. Using the Schrodinger model, —1 acts on the odd functions by —1 and on the even 
functions by +1. Let us include the actions of Z2 C GLi(M) in the model px, consequently in p±. 
Now again, by Mackey analysis, there are four irreducible unitary representations of GLi{M.)Hn-i 
on which G(i?„_i) acts nontrivially, namely 

p±(g)sgn, p±. 

The difference between the former and the latter is a little subtle. One way to tell the difference is 
that {p± (g) sgn)(— 1) acts on the even functions by —1 while p±(— 1) acts on the even functions by 
identity. 

Now consider GLi (M)iJ„_i . The representation p± can be regarded as a representation of Gii (K)i/„_i . 
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Lemma 2.2. Identify GLi{M.)Hn-i with {■^'Z)AHn-i. Then the irreducible unitary representations 
on which C(iJ„_i) act nontrivially are all of the form 

p±®xt\te [0, 1) 

with Xf (to) = exp47ri?Tit for to, G iZ. 

For GLi(]R)i7„_i, t = 0, i because Xi(Z) = 1- 

Now let us consider Sp2n-2i^)- This group preserves p\. Again, by the Stone-Von Neumann 
Theorem, Sp2n-2^) ^cts on p\ projectively. Since 5'p2„_2(^) is already simply connected, one 
obtains a group action of Sp2n-2^) o^i Pa- By a theorem of Segal-Shale- Weil, 5*^2^-2 (^) action on 
px descends into an action of Mp2n-2(K). Simply put, to, S Z acts by (—1)'" = cxpimTr on px. We 
can now extend p± to include the action of 5p2ri-2(^^)- 

Prom now on p± will be representations of P. 

Theorem 2.1 i |Howe| ) . Irreducible unitary representations of P on which C{Hn~i) acts nontrivially 
are of the form 

[P± <»Xt]<»T 

with T an irreducible unitary representation o/5p2n-2(^) such that T{m) — Xti'm-) for any to, G Z. 
In addition, two such representations are equivalent if and only if all the parameters (±,t, r) are the 
same. 

I shall make some remarks here. First, t is extended to a representation of P, trivially on 
AHn-i, and trivially on the component group. Second, xt 'SS t is a. twisted tensor product in 
the sense that the action of Z commutes with the tensor. So Xt("^) ^ '''("■) = Xtin) (8 t(to,) for 
any TO,ri G Z. For group P, p±(rn) (g) Xti'nT') '^ t{0) must be the identity for every m (^ Z Q G. 
So (— l)™exp47rTOi = (— 1)'"t(to,) — 1. For an irreducible unitary representation of P on which 
C(if„_i) acts nontrivially, t — j,j and r is a genuine unitary representation of Mp2n-2(^)- 

The proof is straight forward by applying the Mackey analysis. Observe that the subgroup of 
P that preserves p± is ^Z x^ Sp2n-2(^)- Xt®T parametrizes the equivalence classes of irreducible 
unitary representations of this subgroup. 

3. Irreducible Unitary Representations of Sp2{^) 

Throughout this section G = 5^2 (K) and P = MAN where M ^ Z2 and A ^ R+ and A^ g^ R. 
What we have proved in the last section needs to be modified. Since the unitary dual of G is known 
( |Puk| . |HT| ). we will analyze 7r|p in detail. The results in this section must have been known to 
the experts. They will be used in the next section to analyze higher rank case. 

Fix the standard maximal compact group 5*0(2). We parametrize it by the angle of rotation coun- 
terclockwise. There are essentally four classes of irreducible unitary representations of G (see |Pukj 
ESI): 

(1) the trivial representation 1; 

(2) unitary principal series /(e, s) where e G [0, 1) and s G iK ( we exclude e = |, s = 0); 

(3) complementary series C(e, s) (e G [0, 1), s G (0, |1 — 2e|)); 

(4) Highest weight modules D^ {I > 0) and lowest weight modules D^{1 > 0). 
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Let P be the standard upper triangular parabolic subgroup of G. Let N be the nilradical of P. Then 
the identity component Pq has two irreducible unitary representations on which TV acts nontrivially, 
namely p+ and p_. /9_|_|7v is supported on R+ C N, and P-\n is supported on R~ C N. Now 
the center of G can be identified with ^Z. Identifying P with ^TjAN, p± can be extended to a 

representation of P by identify Pq with P/^Z. Then elements of P are parametrized by (±, t) with 
t € [0,1). More precisely, every irreducible unitary representation of P is equivalent to p±'Si Xt- 
Here xtiinin) = expAirimt with m € ^Z, a € A and n £ N. Notice that Xt also defines a central 
character of G. In our setting, the representations with even weights have central character xo] the 
representations with odd weights have central character xi- Let xi = Xo- 

The following theorem gives the structure of the restriction of irreducible unitary representations of 
G to P. 

Theorem 3.1. Let [-^l be the largest integer less or equal to ^. 

(1) /(e,s)|p^(p+®p_)®Xe; 

(2) G{e,s)\p^(p+®P-)(E)Xe; 

(3) D+\p = p+(^x^-i^i] 

(4) D-|p = p_®Xi-i+Lij- 

The results here are obviously known to the experts. We will provide an elementary proof. 

Proof: The central character of each tt G G can be computed easily. Using the noncompact 
model, /(e, s) can be modeled on L'^{N) with A^ act as translations. Hence (1) is proved. To prove 
(2), (3), (4), it suffices to show that the Fourier transform of the matrix coefficients of it restricted 
to N has the desired support. 

To show (2), let ( , ) be the inner product of G(e, s) and ( , )i,iv be the natural complex linear pairing 
between the induced representations /°°(e, — s) and /°°(e, s). For smooth vectors (jijip E /°°(e, s), 
we have 

(0,'i/;) = {A{e,s)(j),ip)inv 
where A{e, s) is the intertwining operator defined over smooth vectors. In addition, A(e, s) defines 
a bijection between /°°(e, s) and /°°(e, — s). Using the noncompact model, for every n G A^ as an 
additive group, we have 



(G(e, s){n)(l), -0) = (A(e, s)(t>, /(e, s)(-n)?A)inv = / A{e, s)4){x)%l){x + n)dx. 

J N 

Now C^iN) C /°°(e,±s). We choose and if; so that 

A{e,s)^eC^{N), i^eC^{N). 

So (G(e, s){n)(j)^ ■0) becomes the convolution of two smooth and compactly supported functions. Its 
Fourier transform can be made to be supported on N , upon proper choices of (j) and ^. So again we 
have G(e, s)\p = {p+ ® /?_) ® Xe- 

We will now prove (3). (4) follows immediately from (3). Let I > 0. Notice that D^ is a sub- 
quotient of /(e, I — 1) and of /(e, I — I) with e = ^ — [^J . Let wj+p be of weight I + p in D^ where p 
is a nonnegative even integer. We stick with the noncompact picture. Let 

1 .i,l-xi i±p (l + x^)? 



Va;2 + 1 A + xi' {l + xiY+P 
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be a function in the noncompact model of /(e,/ — 1). Here we choose the standard arg function 
between —^ and ^ to define (1 ± cci)' ii I is not an integer. Let 

be a function in the noncompact model of /(e, 1 — V). Then 

{D^{g)vi+p, vi+q) = C{I{e, I - l)(g)(/);+p, Vi+g)inv {g eG,p>0,q< 0). 
In particular, for n G A^ = M, {D'^{ri)vi+p,vi+q) — C Jj^ (f>i^p{x — n)ipi+q{x)dx. 

Recall that 

(See Ch. 8.3 |Fo) ) . It follows by Fourier analysis that 



where W{^) is a linear combination of derivatives of (exp— ^) ^'+2~^|r+ and W{£,) is supported on 
R+. Therefore 

4>i+p{x - n) ^ I W{(,) exp(-ix^) exp(in^) d^. 

■/R+ 

Even though ipi^q{x) are not in i^, , ^ i-^^^ ipi^q{x) will be in L^. So Fourier transform of ipi+q{x) 
will be a Co function on M multiplied by a monomial of ^. It is easy to see that the Fourier transform 
of {D'^{n)vi+p, vi+q) is supported on M+. Since {vi+p | p > 0} is an orthogonal basis for D^ , D'^\n 
is supported on M+. D 

We shall remark that Theorem 13.11 depends on the parametrization of K and A^. If one chooses 
the opposite parabolic P, then the statements in (3) and (4) will change. There may be other proofs 
to Theorem 13. II The proof of (3) and (4) we give here is more self-contained. 

4. Restriction of Unitary Representations and Irreducibility 

Now let TT be a nontrivial irreducible unitary representation of G. Then 7r|p can be decomposed 
into a direct integral of /9± (8" Xt '8) t. In particular, one can write 

TT = /9+ (g) T+(7r) ffi p_ (g) T_(7r) ® Tq. 

Here t+{tt) and T_(7r) are unitary representations of G'ii(R)5p2„-2(^)- The following theorem says 
that To does not occur. 

Theorem 4.1 (See [Howe]). Let n be a nontrivial irreducible unitary representation of G. Then 
there exist two unitary representations T+(7r) and T_(7r) of GLi{W)Sp2n-2i^) such that 

7r|p = p+ (g)T+(7r) ® p_ ® T_(7r). 

Notice that one of t±{tt) could be zero. This theorem was proved by Howe in [Howe] Pg. 249 for 
the metaplectic group. 

Proof: We will have to prove that 7r|p does not have any subrepresentation on which C(-ff„_i) 
acts trivially. Suppose otherwise, let u be a nonzero vector fixed by C{Hn-i). Let Go be the 
subgroup of G that commutes with Sp2n-2^)- So Go — Sp2{R). Notice that G(7?„_i) C Go and 
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A C Go- Let H be the Hilbert space spanned by TT{go)v for go E Gq. Clearly, H decomposes into a 
direct integral of irreducible unitary representations of Go on which Z acts as a character. Indeed, 
all factorial representations of Go are direct sum of irreducible representations. Now let 

Go 

where Vg € Hg <E) Vg, Hg G Go and dim 14 = m{Hg,H). Then G(if„_i) must fix Vg for almost all s 
with respect to /i. If Hg is not trivial, Hg ^ Vg has no vector fixed by G(i/„_i). Hence, H must be a 
direct sum of the trivial representation of Go. In particular, tt must descend to a representation of 
G. The matrix coefficent g — ^ {Tr{g)v,v) violates the Howe-Moore vanishing Theorem ( [HMj ) . We 
reach a contradiction. D 

Now we can prove one of our main results. 

Theorem 4.2. Let tt be a nontrivial irreducible unitary representation of G such that Z acts by 
exjp2'Kmt (Vto G Zj for a fixed t G [0, 1). Suppose that 7r|p is irreducible. Then n must be a highest 
weight module or a lowest weight module. 

Proof: Let us fix the standard maximal compact group U{n) C G. Then U{n) D S'p2n-2(K) = 
U{n — 1). As usual, the complexified Lie algebra gc decomposes into a direct sum 

Suppose that vrjp is irreducible. By the last Theorem, either T+(7r) or T_(7r) must be zero. Without 
loss of generality, suppose that 7r|p = p+ (g) T+(7r). Notice that p+|c(if„_i) is supported on R+. 
So tt\c(h„~i) must also be supported on R+. Let Go be the subgroup of G that commutes with 
Sp2n-2i^)- Consider the restriction 7r|g, . 7r|(=, can be decomposed into a direct integral of irre- 
ducible unitary representations with multiplicities. By Theorem 13. 1[ among the irreducible unitary 
representations of Go, only the lowest weight modules are supported on 1R+ C C{H„-i). Hence only 
the lowest weight modules occur in the direct integral decomposition of ttJ^ . 

Now tt\q is a direct integral of lowest weight modules. Let f/(l) = Gq n U{n). Then ttJ^^-, 
can only have positive weights. Fix a maximal torus T 13 C/(l) in U{n). ttJ^^ can only have positive 
weights, since the weight space of TT\f is invariant under the Weyl group of U{n). Let v\ be a vector 
with weight A such that ^ A^ is minimal among all possible weights occuring in 7r|^. Notice that 
the set of all possible J^^i is a discrete set in R+. So a minimal J^'^i must exist. Now TT\if/„^ 
must contain an irreducible representation V^ with '^ fii — '^ Xi. Clearly, p~ act on V^ by zero. So 
the module generated by V^ must have a lowest weight module as its quotient. Since n is already 
unitary and irreducible, tt must be a lowest weight module. Now we have shown that tt is a unitary 
lowest weight module. D 

We shall remark that the last paragraph is true even one assumes that the weights for ttJ^^^n is 
bounded from below. 

5. Some Criterions for Lowest Weight Modules 

In this section, we give some characterization of lowest weight modules in terms of their restric- 
tions on certain subgroups. Some of them are well-known to the experts. Let us fix a complex 
structure and an inner product ( , ) on the symplectic space M^" such that the symplectic form 
coincides with the imaginary part of (, ). Let ei,e2,...,e„ be the standard basis over C. Let 
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G = S'p2n(M). Let P be the subgroup preserving Rei. Then P = G'Li(M)5'p2«-2(M)iJ„_i. Let U{n) 
be the subgroup preserving ( , ) . 

Theorem 5.1. Let T he the standard maximal torus in the maximal compact group U{n) of G. Let 
Sp2{^) be the subgroup of G preserving Cei and acting on the complex linear span of {e2, 63, ... , e„} 
by identity. Let U{1) — Sp2(JS.) H U(n). Let tt be a nontrivial irreducible unitary representation of 
G. Let Q be a m,axim,al parabolic subgroup of G. Let N be its nilradical and ZN be the center of N. 
Suppose that ZN 15 C(iJ„_i). The following are equivalent: 

(1) T^lzN 'is supported on a subset of the positive semidefinite cone of ZN, regarded as the space 
of symmetric matrices; 

(2) 'n:\c(H„-i) is supported on M+ C C(if„_i); 

(3) TTJg- ,„, decomposes into a direct integral of lowest weight modules; 

(4) 7r|jj^% only has positive weights; 

(5) 7r|j; only has positive weights; 

(6) the weights of 'iT\jj,-y-, is bounded from below; 

(7) there is an integer k such that every weight A of Tr\f satisfies Xt > k for every i; 

(8) TT is a unitary lowest weight module of G; 

(9) If n = 1, TT is a unitary lowest weight module; ifn>2, 7r|p decomposes into p-i- (g) t+ (tt) and 
the weights ofT-^-{TT)\fr^^ are bounded from below. Here T(l) is the one dimensional compact 
torus in U{n) D Sp2n-2{^) fixing all vectors in the complex span of {ei, 63, ... , e„}. 



Proof: When n — 1, our theorem foUows from Theorem 13. II Suppose now n > 2. 

By [Howej . Tr\zN is supported on GL-orbits on ZN. (1) o (2) is a matter of matrix analysis. 
(2) -^ (3) — > (4) is proved in Theorem 14.21 (4) ^- (3) — > (2) is easier than the other direction. So 
(1), (2), (3) and (4) are equivalent. 

(4) -^ (5) -^ (6) -^ (7) is trivial. (7) -^ (8) follows as in Theorem g^l (8) -^ (6) is also obvi- 
ous. So (8), (7), (6) are equivalent. 

To prove (8) — )■ (9), suppose tt is a nontrivial unitary lowest weight module. By Cor. 14.1] 7r|p = 
p+ (g) T+(7r) p_ (X) T_(7r). If T_(7r) ^ 0, fix a r(l)-eigenvector v with weight A. By tensoring with 
vectors in p_, we obtain T(l)-eigenvector with arbitarily low weight. By Weyl group action, we 
obtain C/(l)-eigenvectors with arbitrarily low weight. This contradicts (6). So tt ^ p^ (E) T+(7r). 
Similarly, weights for T+(7r)|j,.-^x must also be bounded below. Hence (8) — > (9). 

Suppose that (9) holds. So the weights of T+{T^)\f(i) are bounded from below. Let G(2) be the 
subgroup of S'p2n-2(K) that fixes all vectors in the complex linear span of {ei, 63, 64, . . . e„}. So 
T(l) is maximal compact in G(2) and t^{t:)\^ — must be a direct integral of lowest weight modules 
with multiplicites. So the weights of T-^-{^:)\f,^^ must all be positive. Hence the weights of Trlj^^-, 
must all be positive. Due to the action of the Weyl group of U{n), 7^|j;^^ only has positive weights. 
So (9) -^ (4). We have proved (9) -> (4) -^ (5) -^ (6) -^ (7) ^ (8) ^ (9). D 

We shall remark here that the parametrization of unitary highest or lowest weight modules is al- 
ready known, due to the work of Enright-Howe-Wallach ( [EH W' ) . It is perhaps easy to go further 
to derive more properties of unitary lowest or highest weight modules from Theorem 15. II 
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Corollary 5.1. Let tt be an irreducible unitary lowest weight module of G. Then 7r|p = p+ (S)T+{tt) 
and T-^{tt) decomposes into a direct sum of lowest weight modules o/ 5*^2,1-2 (^)- 

Proof: By Theorem 15.11 (9), the weights of T+(7r)|y^(^-^-, are bounded from below. By Theorem 
15.11 (4), r_|_(7^)|JJ^^ only has positive weights. By Theorem 15.11 (7) (8), there is a lowest weight 

subrepresentation of Sp2n-2{^) i^i '''+M- Consider the orthogonal complement. If it is nonzero, 
then there is another lowest weight subrepresentation. This process can continue and it will end in 
countable time due to the fact that tt has a countable basis. We now obtain a discrete decomposition. 

D 



6. The group U{p,q) 

Suppose p > q > 1 and p + q > 3. Let U{p, q) be the group that preserve a Hermitian form ( , ) on 
CP+9 with signature (p, q). Let P be a maximal parabolic subgroup that preserves a one dimensional 
isotropic subspace. Then P can be identified with GLi{€.)U{p — l,q— l)iJp+q_2- Here Hp^q^2 are 
parametrized by (i G R, w S C^"'"''"^) . The adjoint action of g G U{p — 1, g — 1) on Hp+q^2 leaves 
t fixed and operates on u as the left multiplication. The adjoint action of a G GLi(C) on _ffp_|_g_2 
dilates t to ||a|pi and operates on u as scalar multiplication. Write G'Li(C) = AU{1) where A — R+. 

Let G ^ {{g,t) \ g Cz U{p,q),e:xjp2T:it = dctg} be an infinite covering of G. For any subgroup 
H of G, let H be the preimage of H. Let p± be the irreducible unitary representations of AHp+g-2 
we defined earlier. These are the only irreducible unitary representations on which C(iJp+g_2) acts 
nontrivially Extend p± to a unitary representation of P. Notice that P+li/(i-)j7(„_i „_i) decomposes 
as follows 

[e„ezdet"+'^ ®0(det"+'^)] ®C°°. 

Here det"'^^ is a character of f7(l), 6'(det"+") is the theta lift of det"+" with respect to 
(f/(l), U{p -l,q~ 1)) (see [EV] [g [E3) and C°° records the multiphcity 

Theorem 6.1. Let n be an infinite dimensional irreducible unitary representation of G. Then there 
are two unitary representation r±(7r) ofU{p-' l,q~l) such that 

tt\p = /9+ ® T+(7r) ©p_ (g)T_(7r). 

Let U{1) be diagonally embedded in U{1, 1) C U{p,q). Then 

^\u(i)u(v-ia-i) =[®nezdet"+'^«)6l(det"+'^)]®r+(7r)(g)C°° 



,^. \U{l)U{p-l,q-l) 



sdet"+— (»0'(det"+— )] (^T.W 



Here 9' refers to the theta lifts with respect to the contragredient oscillator representation. 
Notice that one of the t±{t:) could be zero. Similarly we can prove 

Theorem 6.2. Let vr be an infinite dimensional irreducible unitary representation of G. Suppose 
that 7r|p is irreducible. Then n must be a highest weight module or a lowest weight module. 

Essentially SU{1, 1) in SU{p,q) will play the role of 5p2(R) in 5p2n(R)- The proof is omitted 
here. 
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7. The group 0*{2n) 

Let n > 3. Let 0*{2n) be the group of isometry preserving a nondegenerate skew-Hermitian 
form on H". Let P be the maximal parabohc subgroup preserving a 1 dimensional isotropic sub- 
space. Then P can be identified with GLi{M)0*{2n — 4)_ff2n-4 where H2n-4 is a Heisenberg group 
parametrized by 

GLi{M.) can be further decomposed as Sp{l)A where A is the center of GLi{M). The adjoint action 
of a S A on H2n-4 is given by 

{t, u) -^ (a t, au). 

The adjoint action of 0*{2n — 4) on i?2n-4 is the left multiplication on u. The adjoint action of 
k e Sp{l) on H2n-4 is the right multiplication. Clearly, Sp{l) x 0*{2n — 4) action preserves the 
real part of canonical skew-Hermitian form on H"^^. {Sp{l), 0*{2n — 4)) becomes a dual reductive 
pair (See jH]). 

Now let p± be the two irreducible unitary representations of AH2n-i on which C(i?2ri-4) acts 
nontrivially. p± extends to irreducible unitary representations of the linear group P. In particular, 
P+\sp(i)o* (2n-4:) decomposes according to the theta correspondence with infinite multiplicity: 

Similarly, 

p-lsp(i)o*(2„-4) = ®^^s;irf ® ^'('^) ® '^°°- 

Here 9' is the theta correspondence with respect to the contragredient oscillator representation. 
Sp{l) is parametrized by N. 

Theorem 7.1. Let n be a nontrivial irreducible unitary representation ofO*(2n). Then there exists 
two unitary representations t±{t:) o/0*(2n — 4) such that 

■k\p = p+ ® T+(7r) © p_ ® T„(7r). 

In addition, 

T^\spii)0'(2n-4) = {(S^^g^^[a (g> 0{a)] ® T+in) (g, C°°} © {©^^^^[a (g> 0'{a)] (g> r_(7r) (g, C°°} 

One of t±{tt) could be zero. The theorem for the universal covering of 0*{2n — 4) is left to the 
reader. Similarly, we have 

Theorem 7.2. Lett: be a nontrivial irreducible unitary representation ofO*{2n). If Tr\p is irre- 
ducible, then TT must be a highest weight module or lowest weight module. 

Notice that the group 0*(4) contains a noncompact factor SU{1,1) = SL2{M.). The proof is 
essentially the same as in Theorem 14.21 
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